Image processing method, device and photographic apparatus

ABSTRACT

An image processing method includes correcting a target image based on an initial distortion coefficient to obtain a first corrected target image, performing straight-line fitting on a first border line in the first corrected target image to calculate a first distortion metric value and a correction distortion coefficient, correcting the target image based on the correction distortion coefficient to obtain a second corrected target image, removing outlier points on a second border line in the second corrected target image, performing straight-line fitting on the second border line with the outlier points removed to calculate a second distortion metric value, detecting whether a preset correction condition is satisfied based on at least one of the first distortion metric value or the second distortion metric value, and, if the preset correction condition is satisfied, applying the correction distortion coefficient to subsequent image correction to obtain better corrected images.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of application Ser. No. 15/617,488,filed on Jun. 8, 2017, which is a continuation application ofInternational Application No. PCT/CN2014/093389, filed on Dec. 9, 2014,the entire contents of both of which are incorporated herein byreference.

FIELD OF THE DISCLOSURE

The present disclosure relates to the technical field of imageprocessing, and in particular, to an image processing method, device,and photographic apparatus.

BACKGROUND

As wide-angle lenses have been more and more widely used, especiallywith their use on aerial shooting apparatuses, camera distortion causedby the wide-angle lenses has attracted more and more attention. Ifcorrection is not made, images or videos taken by the wide-angle lensesmay have a serious barrel distortion. For example, when a sport court isshot, straight lines sprayed on the court may be distorted and appear ascurved lines. Therefore, a camera may need to be calibrated to obtain adistortion coefficient thereof to correct the images or videos taken bythe camera.

Camera distortion generally includes radial distortion and tangentialdistortion. For a wide-angle lens, a fourth-order polynomial radialdistortion model has been proved to be sufficient. Distortion equationsof a wide-angle lens are:

x ^(d) =x ^(u)(1+k ₁ r ² +k ₂ r ⁴)   (1)

y ^(d) =y ^(u)(1+k ₁ r ² +k ₂ r ⁴),   (2)

where (x^(u), y^(u)) denote coordinates before distortion, also referredto as “non-distorted coordinates,” and (x^(u), y^(u)) denotecorresponding coordinates after distortion, also referred to as“distorted coordinates.” A distortion center can be represented by itscoordinates (c_(x), c_(y)), and the parameter r in Equations (1) and (2)can be calculated using r=√{square root over((x^(u)−c_(x))²+(y^(u)−c_(y))².)} The purpose of the calibration is todetermine distortion coefficients k =(k₁, k₂) and the distortion centerc=(c_(x), c_(y)).

Usually, the distortion center (c_(x), c_(y)) in an image is close to acentral point

$\left( {\frac{w}{2},\frac{h}{2}} \right)$

of the image, where w and h are the width and the height of the image,respectively. The central point of the image can be used as an initialvalue of the distortion center, and an optimal solution to thedistortion center can be obtained by a small number of times ofiteration. However, it may be relatively more difficult to calculate thedistortion coefficients.

In conventional technologies, a lens can be calibrated using acalibration board. In particular, a series of images or videos of thecalibration board can be taken, and then internal references anddistortion coefficients can be calculated according to geometricconstraints. After the distortion coefficients are obtained, the shotvideos or images can be corrected. Calibration using the calibrationboard can be relatively accurate as long as a high-accuracy calibrationboard is used. This is a barrier to ordinary users when performingcalibration.

SUMMARY OF THE DISCLOSURE

Embodiments of the present disclosure provide an image processingmethod, device, and video camera, which can easily and rapidly determinea distortion coefficient to process an image.

In accordance with the disclosure, there is provided an image processingmethod including correcting a target image based on an initialdistortion coefficient to obtain a first corrected target image,performing straight-line fitting on a first border line in the firstcorrected target image to calculate a first distortion metric value anda correction distortion coefficient, correcting the target image basedon the correction distortion coefficient to obtain a second correctedtarget image, performing straight-line fitting on a second border linein the second corrected target image to calculate a second distortionmetric value, detecting whether a preset correction condition issatisfied based on at least one of the first distortion metric value orthe second distortion metric value, and configuring the correctiondistortion coefficient as the initial distortion coefficient if thepreset correction condition is not satisfied.

In some embodiments, the method further includes, before correcting thetarget image, capturing an image of an object including straight linefeatures and adjusting a size of the captured image to obtain the targetimage.

In some embodiments, adjusting the size of the captured image includesmagnifying, if the size of the captured image is smaller than a presetsize threshold, the captured image to a target size throughinterpolation, or scaling down, if the size of the captured image isgreater than the preset size threshold, the captured image to the targetsize through down-sampling.

In some embodiments, performing straight-line fitting on the firstborder line to calculate the first distortion metric value and thecorrection distortion coefficient includes performing edge detection onthe first corrected target image to determine the first border line inthe first corrected target image, performing straight-line fitting onthe first border line based on polynomial straight-line fitting toobtain a fitted straight line, and calculating the first distortionmetric value of the first border line relative to the fitted straightline and the correction distortion coefficient corresponding to thefirst distortion metric value.

In some embodiments, calculating the first distortion metric value andthe correction distortion coefficient includes determining a straightline segment in the first border line, calculating distances fromcorresponding points on the straight line segment to the fitted straightline, obtaining the first distortion metric value according to thedistances, and performing non-linear optimization on the firstdistortion metric value to obtain the correction distortion coefficient.

In some embodiments, correcting the target image according to theinitial distortion coefficient includes correcting a target border linein the target image based on the initial distortion coefficient, andcorrecting the target image based on the correction distortioncoefficient includes correcting the target border line in the targetimage based on the correction distortion coefficient.

In some embodiments, performing straight-line fitting on the secondborder line to calculate the second distortion metric value includesperforming edge detection on the second corrected target image todetermine the second border line in the second corrected target image,performing straight-line fitting on the second border line based onpolynomial straight-line fitting to obtain a fitted straight line, andcalculating the second distortion metric value of the second border linerelative to the fitted straight line.

In some embodiments, calculating the second distortion metric valueincludes, removing outliers, determining a straight line segment in thesecond border line, calculating distances from corresponding points onthe straight line segment to the fitted straight line, and obtaining thesecond distortion metric value according to the distances.

In some embodiments, detecting whether the preset correction conditionis satisfied includes calculating a relative variation amount betweenthe first distortion metric value and the second distortion metricvalue, and determining whether the relative variation amount calculatedis smaller than a preset variation threshold to determine whether thepreset correction condition is satisfied.

In some embodiments, the method further including performing imagecorrection based on the correction distortion coefficient if the presetcorrection condition is satisfied.

Also in accordance with the disclosure, there is provided a cameraincluding a camera lens and an image processor. The image processor isconfigured to correct a target image based on an initial distortioncoefficient to obtain a first corrected target image, performstraight-line fitting on a first border line in the first correctedtarget image to calculate a first distortion metric value and acorrection distortion coefficient, correct the target image based on thecorrection distortion coefficient to obtain a second corrected targetimage, perform straight-line fitting on a second border line in thesecond corrected target image to calculate a second distortion metricvalue, detect whether a preset correction condition is satisfied basedon at least one of the first distortion metric value or the seconddistortion metric value, and configure the correction distortioncoefficient as the initial distortion coefficient if the presetcorrection condition is not satisfied.

In some embodiments, the image processor is further configured tocapture an image of an object including straight line features throughthe camera lens, and adjust a size of the captured image to obtain thetarget image.

In some embodiments, the image processor is further configured tomagnify, if the size of the captured image is smaller than a preset sizethreshold, the captured image to a target size through interpolation, orscale down, if the size of the captured image is greater than the presetsize threshold, the captured image to the target size throughdown-sampling.

In some embodiments, the image processor is further configured toperform edge detection on the first corrected target image to determinethe first border line in the first corrected target image, performstraight-line fitting on the first border line based on polynomialstraight-line fitting to obtain a fitted straight line, and calculatethe first distortion metric value of the first border line relative tothe fitted straight line and the correction distortion coefficientcorresponding to the first distortion metric value.

In some embodiments, the image processor is further configured todetermine a straight line segment in the first border line, calculatedistances from corresponding points on the straight line segment to thefitted straight line, obtain the first distortion metric value accordingto the distances, and perform non-linear optimization on the firstdistortion metric value to obtain the correction distortion coefficient.

In some embodiments, the image processor is further configured tocorrect the target image according to the initial distortion coefficientby correcting a target border line in the target image based on theinitial distortion coefficient, and correct the target image based onthe correction distortion coefficient by correcting the target borderline in the target image based on the correction distortion coefficient.

In some embodiments, the image processor is further configured toperform edge detection on the second corrected target image to determinethe second border line in the second corrected target image, performstraight-line fitting on the second border line based on polynomialstraight-line fitting to obtain a fitted straight line, and calculatethe second distortion metric value of the second border line relative tothe fitted straight line.

In some embodiments, the image processor is further configured to removeoutliers, determine a straight line segment in the second border line,calculate distances from corresponding points on the straight linesegment to the fitted straight line, and obtain the second distortionmetric value according to the distances.

In some embodiments, the image processor is further configured tocalculate a relative variation amount between the first distortionmetric value and the second distortion metric value, and determinewhether the relative variation amount calculated is smaller than apreset variation threshold to determine whether the preset correctioncondition is satisfied.

In some embodiments, the image processor is further configured toperform image correction based on the correction distortion coefficientif the preset correction condition is satisfied

According to the embodiments of the present disclosure, a distortioncoefficient of an image can be determined comprehensively based onstraight-line fitting and a distortion metric value. This optimizes adistortion coefficient calculation and can obtain a more accuratedistortion coefficient automatically and intelligently. The methods anddevices consistent with embodiments of the disclosure also do notrequire an additional calibration board, and have a low cost and areeasy for users to use.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic flow chart of one image processing methodaccording to an embodiment of the present disclosure;

FIG. 2 is a schematic flow chart of another image processing methodaccording to an embodiment of the present disclosure;

FIG. 3 is a schematic flow chart of processing an image to obtaindistortion coefficients according to an embodiment of the presentdisclosure;

FIG. 4 is a schematic flow chart of processing a corrected image toobtain distortion coefficients according to an embodiment of the presentdisclosure;

FIG. 5 is a schematic structural diagram of one image processing deviceaccording to an embodiment of the present disclosure;

FIG. 6 is a schematic structural diagram of another image processingdevice according to an embodiment of the present disclosure;

FIG. 7 is a schematic structural diagram of a processing module in FIG.6;

FIG. 8 is a schematic structural diagram of a detection module in FIG.6; and

FIG. 9 is a schematic structural diagram of a video camera according toan embodiment of the disclosure.

DETAILED DESCRIPTION OF THE EMBODIMENTS

The technical solution of the present disclosure will be described inmore detail below with reference to the accompanying drawings. Thedescribed embodiments are merely some of the embodiments of the presentdisclosure rather than all of the embodiments. All other embodimentsobtained by a person of ordinary skill in the art based on theembodiments of the present disclosure without creative efforts shallfall within the scope of the present disclosure.

FIG. 1 is a schematic flow chart of an image processing method accordingto an embodiment of the present disclosure. The image processing methodaccording to the embodiment of the present disclosure may be performedby an image processor. As shown in FIG. 1, at S101, a target image iscorrected based on initial distortion coefficients to obtain a firstcorrected target image, and straight-line fitting is performed on aborder line in the first corrected target image to calculate a firstdistortion metric value and correction distortion coefficients.

The initial distortion coefficients may be pre-configured. In someembodiments, the initial distortion coefficients can be configuredaccording to a model of a camera lens.

The border line may be determined from the target image through edgedetection. In the target image, the border line may be a straight-lineedge of a building, and distortion of the target image can be correctedbased on the border line that should be a straight line.

The edge detection may be based on positions of pixel points andamplitude variations of pixel values of the pixel points. In someembodiments, the edge detection can include a detection method having asub-pixel accuracy.

Simple polynomial straight-line fitting may be employed for thestraight-line fitting of the border line. A series of discrete pixelpoints that are supposed to be on a straight line, e.g., the borderline, may scatter around a straight line in the image due to distortion.These discrete pixel points can be fitted into a straight line. Thefitted straight line is used to reflect a basic trend of these discretepixel points.

The first distortion metric value E₀₁ can be obtained according todistances from the discrete pixel points to the fitted straight line.For example, E₀₁ may be the sum of squares of the distances from thediscrete pixel points to the fitted straight line. The smaller E₀₁ is,the smaller the distortion of the target image is, or vice versa. AfterE₀₁ is obtained, by performing a non-linear optimization on a functioncorresponding to E₀₁, a set of distortion coefficients that minimizesE₀₁ can be determined. The determined distortion coefficients are thusthe correction distortion coefficients. In the following embodiments,one exemplary expression of the function corresponding to a distortionmetric value will be further described in detail.

At S102, the target image is corrected based on the correctiondistortion coefficients to obtain a second corrected target image, andstraight-line fitting is performed on a border line in the secondcorrected target image to calculate a second distortion metric value,E₀₂.

In order to save computing time and computing resources, the correctionof the target image in S102 may include only the correction of theborder line in the target image. When straight-line fitting is carriedout, outliers can be removed to calculate the second distortion metricvalue more quickly.

The second distortion metric may be calculated in a manner similar tothat described above for the first distortion metric.

At S103, whether a preset correction condition is satisfied is detected.

Whether the first distortion metric value and the second distortionmetric value satisfy the preset correction condition can be determinedby judging whether a relative variation amount between the firstdistortion metric value and the second distortion metric value issmaller than a preset variation threshold. In some embodiments, anequation for calculating the relative variation amount may be:(E₀₁−E₀₂)/E₀₂. If a calculated result is smaller than the presetvariation threshold, the correction condition is satisfied. On the otherhand, if the calculated result is not smaller than the preset variationthreshold, the correction condition is not satisfied.

In some embodiments, whether the preset condition is satisfied can bedetermined by judging whether the second distortion metric value issmaller than a preset metric threshold. If the second distortion metricvalue is smaller than the preset metric threshold, it is determined thatthe preset correction condition is satisfied.

At S5104, if the preset correction condition is not satisfied, thecorrection distortion coefficients are configured as the initialdistortion coefficients, and the process returns to S101. Theabove-described processes in S101-S103 can be repeated until the presetcorrection condition is satisfied.

That is, if the preset correction condition is not satisfied, e.g., ifthe relative variation amount is not smaller than the preset thresholdor if the second distortion metric value is not smaller than the presetmetric threshold, then the correction to the target image is notsufficient enough to correct the distortion, and the distortion of thetarget image is still relatively large. Therefore, the target image isfurther corrected according to the correction distortion coefficients,and a new distortion metric value is calculated to determine whether thecorrected target image meets the requirement.

At S105, if the preset correction condition is satisfied, imagecorrection is performed based on the correction distortion coefficients.

That is, if the preset correction condition is satisfied, e.g., if therelative variation amount is smaller than the preset variation thresholdor if the second distortion metric value is smaller than the presetmetric threshold, then the correction to the target image with thecorrection distortion coefficients has met the requirement fordistortion correction. The correction distortion coefficients areoutputted for performing subsequent correction on other target images orother related processing.

The method according to the embodiment of the present disclosure can beperformed when a video camera is being initialized, and the obtainedcorrection distortion coefficients can be stored in a memory in order toperform subsequent processing based on the obtained correctiondistortion coefficients. In some embodiments, the method according tothe present disclosure can be performed every time when shooting isperformed to obtain correction distortion coefficients.

According to the embodiment of the present disclosure, distortioncoefficients of an image can be determined based on straight-linefitting and a distortion metric value. This optimizes the calculation ofthe distortion coefficients and can obtain more accurate distortioncoefficients automatically and intelligently. Further, additionalcalibration board is not required, which reduces cost and is easy forusers to use.

FIG. 2 is a schematic flow chart of another image processing methodaccording to an embodiment of the present disclosure. The imageprocessing method according to the embodiment of the present disclosuremay be performed by an image processor. As shown in FIG. 2, at S201, animage of an object that includes straight line features is captured.

In some embodiments, multiple pictures can be captured and analyzed,among which one or more images including an object with straight linefeatures, such as a building, a playground, or a motorway, can be usedas target images for subsequent distortion analysis. In someembodiments, the multiple images can be processed simultaneously orseparately, and each image is processed in the same manner.

At S202, a size of the captured image is adjusted to obtain a targetimage.

In some embodiments, if the size of the captured image is smaller than apreset size threshold, the captured image can be magnified to a targetsize through interpolation. If the size of the captured image is greaterthan the preset size threshold, the captured image can be scaled down tothe target size through down-sampling.

Because the distortion coefficients are irrelevant to the size of theimage, the size of the target image can be adjusted to balance computingtime and accuracy. If the image is too small, the image is magnified tothe target size through interpolation to improve computing accuracy. Ifthe image is too large, the image is reduced to the target size throughdown-sampling to improve computing speed.

At S203, the target image is corrected based on initial distortioncoefficients to obtain a first corrected target image.

In some embodiments, preset initial distortion coefficients areacquired. In some embodiments, camera model information is detected anddistortion coefficients corresponding to the camera model information issearched. The located distortion coefficients are configured as theinitial distortion coefficients.

That is, a user may directly enter the initial distortion coefficientsaccording to an actual condition of the camera in order to shorten thecomputing time of the distortion coefficients. The initial distortioncoefficients can also be automatically configured based on a model ofthe camera. Usually, distortion coefficients of cameras or lenses of thesame model are about the same and are usually only slightly different.Thus, after the model of the camera or the lens is detected, commondistortion coefficients of the model can be used as the initialdistortion coefficients.

In some embodiments, in the process of correcting the image using theinitial distortion coefficients, it is feasible to merely correct theborder line in the target image to reduce the correction time, therebyshortening the entire computing time for the distortion coefficients.

At S204, a straight-line fitting is performed on a border line includedin the first corrected target image to calculate a first distortionmetric value and correction distortion coefficients. The border line inthe first corrected target image is also referred to as a “first borderline.” Simple polynomial fitting may be employed for the straight-linefitting. Exemplary methods for the straight-line fitting and calculatingthe distortion metric value and the correction distortion coefficientsare described in more detail below.

At S205, the target image is corrected based on the correctiondistortion coefficients to obtain a second corrected target image, and astraight-line fitting is performed on a border line included in thesecond corrected target image to calculate a second distortion metricvalue. The border line in the second corrected target image is alsoreferred to as a “second border line.” In some embodiments, in theprocess of correcting the target image using the correction distortioncoefficients, it is feasible to merely correct a border line in thetarget image to reduce the correction time, thereby shortening theentire computing time for the distortion coefficients.

At S206, whether a preset correction condition is satisfied is detected.This can include, for example, whether a relative variation amountbetween the first distortion metric value and the second distortionmetric value is smaller than a preset variation threshold. If therelative variation amount is smaller than the preset variationthreshold, the correction condition is satisfied. Otherwise, thecorrection condition is not satisfied. In some embodiments, the relativevariation amount can be calculated using: (E₀₁−E₀₂)/E₀₂. If a calculatedresult is smaller than the preset threshold, the correction condition issatisfied. If the calculated result is not smaller than the presetthreshold, the correction condition is not satisfied.

At S207, if the correction condition is not satisfied, the correctiondistortion coefficients are configured as the initial distortioncoefficients, and S203 to S206 are repeated until the preset correctioncondition is satisfied. That is, if the relative variation amount is notsmaller than the preset variation threshold, a relationship between thefirst distortion metric value and the second distortion metric valuedoes not satisfy the preset correction condition. This indicates thatthe correction to the target image using the correction distortioncoefficients still does not meet the requirement for distortioncorrection, and the distortion of the target image is still relativelylarge. The target image is again corrected based on the correctiondistortion coefficients, and a new distortion metric value is calculatedto determine whether the target image meets the requirement.

At S208, if the preset correction condition is satisfied, imagecorrection is performed based on the correction distortion coefficients.

That is, if the relative variation amount is smaller than the presetthreshold and the correction condition is satisfied, or a resultobtained after the above processes are repeated is smaller than thepreset threshold and the correction condition is satisfied, therelationship between the first distortion metric value and the seconddistortion metric value satisfies the preset correction condition. Thisindicates that correction to the target image with the correctiondistortion coefficients has met the requirement for distortioncorrection, and the correction distortion coefficients can be outputtedfor performing subsequent corrections on other target image and otherrelated processing.

FIG. 3 is a schematic flow chart of processing an image to obtaindistortion coefficients according to an embodiment of the presentdisclosure. The method shown in FIG. 3 corresponds to S204 in FIG. 2. Asshown in FIG. 3, at S301, edge detection is performed on the firstcorrected target image to determine the border line in the firstcorrected target image. In some embodiments, a detection method having asub-pixel accuracy may be employed for the edge detection, and candepend on software and hardware resource conditions. For example, in thecase that the computing resources are limited, a general integer-pixeledge detection method can be used to detect the border line.

At S302, straight-line fitting is performed on the determined borderline based on polynomial straight-line fitting. In some embodiments,simple polynomial fitting may be employed for the straight-line fitting.For example, for n points (x_(j), y_(j)) (j=1, 2, 3, . . . , n) on astraight line i, l_(i)=(a_(i),b_(i),c_(i)) can be used to represent thestraight line i corresponding to these points, where x_(j), y_(j),a_(i), b_(i), and c_(i) satisfy the following equation:

a _(i) x _(j) +b _(i) y _(j) +c _(i)=0.

Different methods can be employed to estimate a_(i), b_(i), and c_(i).For example, they can be calculated as follows:

a_(i) = sin  θ, b_(i) = cos  θ, and  ${c_{i} = {{{- \overset{\_}{x}}\; \sin \; \theta} - {\overset{\_}{y}\; \cos \; \theta}}},{{where}\text{:}}$${{\overset{\_}{x} = {\frac{1}{n}{\sum_{j = 1}^{n}x_{j}}}},{\overset{\_}{y} = {\frac{1}{n}{\sum_{j = 1}^{n}y_{j}}}},{and}}\mspace{14mu}$${{\tan \; 2\; \theta} = {- \frac{2V_{xy}}{V_{xx} - V_{yy}}}},{{where}\text{:}}$${{V_{xx} = {\frac{1}{n}{\sum_{j = 1}^{n}\left( {x_{j} - \overset{\_}{x}} \right)}}},{V_{xy} = {\frac{1}{n}{\sum_{j = 1}^{n}{\left( {x_{j} - \overset{\_}{x}} \right)\left( {y_{j} - \overset{\_}{y}} \right)}}}},{and}}\mspace{14mu}$$V_{yy} = {\frac{1}{n}{\sum_{j = 1}^{n}{\left( {y_{j} - \overset{\_}{y}} \right).}}}$

As such, a_(i), b_(i), and c_(i) can be estimated using the aboveequations.

Another method of estimating a_(i), b_(i), and c_(i) is described below.Assume matrix X is set as a matrix formed by homogenous expressions ofthe above-mentioned set of n points, i.e.,

$X = {\begin{bmatrix}x_{1} & y_{1} & 1 \\x_{2} & y_{2} & 1 \\\ldots & \ldots & \ldots \\x_{n} & y_{n} & 1\end{bmatrix}.}$

In an ideal scenario, all points (x_(j), y_(j)) are on a same linerepresented by l, i.e.,

Xl=0.

However, since the points are not ideal, Xl may only be approximatelyequal to 0. That is, an optimal solution of l can be used as thestraight line i corresponding to the points (x_(j), y_(j)). The optimalsolution of l satisfies

$\min\limits_{l}{{{Xl}}^{2}.}$

Various methods can be used to solve this optimization problem. Forexample, the expression of l can be obtained by solving the followingoptimization equation:

${\min\limits_{l}{{Xl}}^{2}},{{s.t.\mspace{11mu} {l}} = 1.}$

The solution of this equation is a right singular vector correspondingto the smallest singular value of X Suppose that the singular value of Xis decomposed as X=UΣV^(T), then l=V₃, where V₃ is the third column ofV, i.e., the right singular vector corresponding to the smallestsingular value. In order to facilitate calculation of a distance from apoint to the straight line, after the straight line coefficients areobtained, the coefficients can be multiplied by a scaling factor suchthat a_(i) ²+b_(i) ²=1.

In addition, in order to minimize the impact of outliers on thestraight-line fitting, fitting points can be selected by using RANSAC(RANdom SAmple Consensus).

At S303, the first distortion metric value of the border line relativeto the straight line corresponding to the above-described straight-linefitting and the correction distortion coefficients corresponding to thefirst distortion metric value are calculated. The straight linecorresponding to the straight-line fitting is also referred to as a“fitted straight line.” In some embodiments, a straight line segment inthe border line can be determined and distances from correspondingpoints on the straight line segment to the fitted straight line can becalculated. The first distortion metric value can be obtained accordingto the calculated distances. The correction distortion coefficients canbe obtained by non-linearly optimizing the first distortion metricvalue.

In some embodiments, multiple fitted straight lines can be obtained bythe straight-line fitting. Adjacent straight lines having closecoefficients can be connected to form one straight line, and the firstdistortion metric value can be calculated for all of the fitted straightlines, which can be the sum of squares of distances from the points tothe lines:

E ₀₁=Σ_(i)Σ_(jΣl) _(i) (a _(i) x _(j) +b _(i) y _(j) +c _(i))².

where a_(i), b_(i), and c_(i) are coefficients of the i-th straight lineamong the multiple straight lines. The smaller E₀₁ is, the smaller thedistortion is. The greater E₀₁ is, the greater the distortion is.

Moreover, it can be seen from the above equation that the distortionmetric is a function of x_(j) and y_(j). In the distortion equation,x_(j) and y_(j) are equations for the distortion coefficients k=(k₁,k₂). Therefore, the distortion metric is a function of the distortioncoefficients. A set of k₁, k₂) values can be found by non-linearlyoptimizing the function corresponding to the distortion metric thatminimizes the distortion metric value, and the set k=(k₁, k₂) are thecorrection distortion coefficients.

FIG. 4 is a schematic flow chart showing a method of processing acorrected image to obtain distortion coefficients according to anembodiment of the present disclosure. The process shown in FIG. 4corresponds to S205 in FIG. 2. As shown in FIG. 4, at S401, edgedetection is performed on the second corrected target image to determinethe border line in the second corrected target image.

At S402, straight-line fitting is performed on the determined borderline based on polynomial straight-line fitting.

At S403, the second distortion metric value of the border line relativeto a straight line corresponding to the straight-line fitting iscalculated. In some embodiments, calculating the second distortionmetric value can include removing outliers, determining a straight linesegment in the border line, calculating distances from correspondingpoints on the straight line segment to the straight line correspondingto the straight-line fitting, and obtaining the second distortion metricvalue according to the calculated distances.

Reference can be made to the description of the corresponding embodimentof FIG. 3 for the processes of the edge detection, the straight-linefitting, and the calculation of the second distortion metric value.

According to the embodiments of the present disclosure, distortioncoefficients of an image can be determined based on both straight-linefitting and a distortion metric value. This optimizes a distortioncoefficient calculation and can obtain more accurate distortioncoefficients automatically and intelligently. The methods consistentwith embodiments of the disclosure do not require an additionalcalibration board, and have a low cost and are easy for users to use.

An image processing device and a video camera according to theembodiments of the present disclosure are described below in detail.

FIG. 5 is a schematic structural diagram of an image processing deviceaccording to an embodiment of the present disclosure. The deviceaccording to the embodiment of the present disclosure can be configuredin various kinds of video cameras. As shown in FIG. 5, the imageprocessing device includes a processing module 1, a detection module 2,and a correction module 3.

The processing module 1 is configured to correct a target image based oninitial distortion coefficients to obtain a first corrected target imageand to perform straight-line fitting on a border line included in thecorrected target image to calculate a first distortion metric value andcorrection distortion coefficients. The processing module 1 is furtherconfigured to correct the target image according to the correctiondistortion coefficients to obtain a second corrected target image and toperform straight-line fitting on a border line included in the secondcorrected target image to calculate a second distortion metric value.

The detection module 2 is configured to detect whether a presetcorrection condition is satisfied. If the preset correction condition isnot satisfied, the detection module 2 configures the correctiondistortion coefficient as the initial distortion coefficient, andforwards the new initial distortion coefficient to the processing module1 such that the processing module 1 can repeat the correction andstraight-line fitting processes based on the new initial distortioncoefficient.

The correction module 3 is configured to, when a detection result of thedetection module 2 is that the preset correction condition is satisfied,perform image correction based on the correction distortioncoefficients.

The initial distortion coefficients used in the image correctionperformed for the first time by the processing module 1 may bepre-configured. In some embodiments, the initial distortion coefficientscan be configured according to a model of a camera lens.

The processing module 1 may determine the border line from the targetimage by way of edge detection. In the target image, the border line maybe a straight-line edge of a building, and distortion of the targetimage can be corrected through the border line that should be a straightline.

The edge detection used by the processing module 1 may be based onpositions of pixel points and amplitude variations of pixel values ofthe pixel points. In some embodiments, the edge detection can include adetection method having a sub-pixel accuracy.

Simple polynomial straight-line fitting may be employed for thestraight-line fitting on the border lines by the processing module 1. Aseries of discrete pixel points that are supposed to be on a straightline, e.g., the border line, may scatter around a straight line in theimage due to distortion. These discrete pixel points can be fitted intoa straight line. The fitted straight line is used to reflect a basictrend of these discrete pixel points.

The processing module 1 can obtain the first distortion metric valueaccording to distances from the discrete pixel points to the fittedstraight line. After the first distortion metric value is obtained,through a non-linear optimization on a function corresponding to thefirst distortion metric value, a set of distortion coefficients thatminimizes the first distortion metric value can be determined. Thedetermined distortion coefficients are thus the correction distortioncoefficients.

In order to save computing time and computing resources, when theprocessing module 1 corrects the target image according to the firstdistortion coefficient, the processing module 1 may only correct theborder line in the target image. When straight-line fitting is carriedout, outliers can be removed to calculate the second distortion metricvalue more quickly.

The detection module 2 may determine whether the first distortion metricvalue and the second distortion metric value satisfy the presetcorrection condition by judging whether a relative variation amountbetween the first distortion metric value and the second distortionmetric value is smaller than a preset variation threshold. In someembodiments, an equation for calculating the relative variation amountmay be: (E₀₁−E₀₂)/E₀₂, where E₀₁ denotes the first distortion metricvalue and E₀₂ denotes the second distortion metric value. If acalculation result is smaller than the preset variation threshold, thecorrection condition is satisfied. On the other hand, if the calculationresult is not smaller than the preset variation threshold, thecorrection condition is not satisfied.

If the detection module 2 detects that the preset correction conditionis not satisfied, e.g., when the relationship between the firstdistortion metric value and the second distortion metric value is notsmaller than the preset variation threshold, the correction to thetarget image is not sufficient enough to correct the distortion, and thedistortion of the target image is still relatively large. In thisscenario, the detection module 2 can notify the processing module 1 tocorrect the target image once again based on the correction distortioncoefficients and to calculate a new distortion metric value to determinewhether the corrected target image meets the requirement.

If the correction condition is satisfied, e.g., if the above relativevariation amount is smaller than the preset threshold or if the seconddistortion metric value is smaller than a preset metric threshold, thedetection module 2 determines that the preset correction condition issatisfied, which indicates that the correction to the target image withthe correction distortion coefficients has met the requirement fordistortion correction. The correction module 3 uses the correctiondistortion coefficients for subsequent image correction and otherrelated processing.

According to the embodiment of the present disclosure, distortioncoefficients of an image can be determined based on straight-linefitting and a distortion metric value. This optimizes a distortioncoefficient calculation and can obtain more accurate distortioncoefficients automatically and intelligently. Further, additionalcalibration board is not required, which reduces cost and is easy forusers to use.

FIG. 6 is a schematic structural diagram of another image processingdevice according to an embodiment of the present disclosure. The deviceaccording to the embodiment of the present disclosure can be configuredin various kinds of video cameras. As shown in FIG. 6, the imageprocessing device includes the processing module 1, the detection module2, the correction module 3, an image acquisition module 4, and a sizeadjustment module 5.

The image acquisition module 4 is configured to capture an image of anobject that includes straight line features. The size adjustment module5 is configured to adjust the size of the captured image to obtain atarget image.

In some embodiments, the size adjustment module 5 is configured to, ifthe size of the captured image is smaller than a preset size threshold,magnify the captured image to a target size through interpolation; and,if the size of the captured image is greater than the preset sizethreshold, scale down the captured image to the target size throughdown-sampling.

The image acquisition module 4 may analyze multiple pictures capturedand use an image including an object with straight line features, suchas a building, a playground, or a motorway, as a target image forsubsequent distortion analysis. In the embodiment of the presentdisclosure, the size adjustment module 5 can process multiple imagessimultaneously or subsequently, and each image can be processed in thesame manner.

Because the distortion coefficients are irrelevant to the size of theimage, the size adjustment module 5 can adjust the size of the targetimage in order to balance computing time and accuracy. If the image istoo small, the size adjustment module 5 magnifies the image to a targetsize through interpolation to improve the computing accuracy; and if theimage is too large, the size adjustment module 5 reduces the image tothe target size through down-sampling to improve the computing speed.

In some embodiments, as shown in FIG. 7, the processing module 1includes a first processing unit 11, a first determination unit 12, acorrection unit 13, a second processing unit 14, and a seconddetermination unit 15.

The first processing unit 11 configured to perform edge detection on thetarget image to determine a border line in the target image, and toperform straight-line fitting on the determined border line based onpolynomial straight-line fitting to obtain a fitted straight line.

The first determination unit 12 is configured to calculate a firstdistortion metric value of the border line relative to the fittedstraight line and correction distortion coefficients corresponding tothe first distortion metric value.

In some embodiments, the first determination unit 12 is configured todetermine a straight line segment in the border line, calculatedistances from corresponding points on the straight line segment to thefitted straight line, obtain the first distortion metric value accordingto the calculated distances, and non-linearly optimize the firstdistortion metric value to obtain the correction distortioncoefficients.

The correction unit 13 is configured to correct the border line in thetarget image according to the initial distortion coefficients or thecorrection distortion coefficients to complete correction of the targetimage.

The second processing unit 14 is configured to perform edge detection onthe corrected target image to determine a border line in the correctedtarget image, and perform straight-line fitting on the determined borderline based on polynomial straight-line fitting to obtain a fittedstraight line.

The second determination unit 15 is configured to calculate a seconddistortion metric value of the border line corresponding to the fittedstraight line. In some embodiments, the second determination unit 15 isfurther configured to remove outliers and determine a straight linesegment in the border line, and calculate distances from correspondingpoints on the straight line segments to the fitted straight line andobtain the second distortion metric value based on the calculateddistances.

In some embodiments, as shown in FIG. 8, the detection module 2 includesa variation calculation unit 21 and a condition determination unit 22.The variation calculation unit 21 is configured to calculate a relativevariation amount between the first distortion metric value and thesecond distortion metric value. The condition determination unit 22 isconfigured to, if the relative variation amount calculated is smallerthan a preset variation threshold, determine that the correctioncondition is satisfied, or otherwise, determine that the correctioncondition is not satisfied.

Referring again to FIG. 6, in some embodiments, the device according tothe embodiment of the present disclosure further includes an acquisitionmodule 6 configured to acquire preset initial distortion coefficients,or detect camera model information, search for distortion coefficientscorresponding to the camera model information, and configure the locateddistortion coefficients as the initial distortion coefficients.

Functions of various modules and units in the embodiments describedabove in connection with FIGS. 5-8 are similar to the methods in theembodiments described above in connection with FIGS. 1-4, and detaileddescription thereof is omitted.

According to embodiments of the present disclosure, distortioncoefficients of an image can be determined based on straight-linefitting and a distortion metric value. This optimizes a distortioncoefficient calculation and can obtain more accurate distortioncoefficients automatically and intelligently. Therefore, additionalcalibration board is not required, which reduces cost and is easy forusers to use.

FIG. 9 is a schematic structural diagram of a video camera 900consistent with embodiments of the disclosure. The video camera 900includes a camera lens 910 and an image processing device 920. The imageprocessing device 920 includes an image processor 922 and a memory 924.The memory 924 stores an image processing program containinginstructions consistent with embodiments of the disclosure. When theimage processing program is executed by the image processor 922, itcauses the image processor 922 to execute a method consistent withembodiments of the disclosure, such as one of the exemplary methodsdescribed above.

In some embodiments, the image processor 922 is configured to correct atarget image according to initial distortion coefficients to obtain afirst corrected target image and perform straight-line fitting on aborder line included in the first corrected target image to calculate afirst distortion metric value and correction distortion coefficients.The image processor 922 is further configured to correct the targetimage according to the correction distortion coefficients to obtain asecond corrected target image and perform straight-line fitting on aborder line included in the second corrected target image to calculate asecond distortion metric value. The image processor 922 can detectwhether a preset correction condition is satisfied. If the presetcorrection condition is not satisfied, the image processor 922 canconfigure the correction distortion coefficients as the initialdistortion coefficients, and perform processing again until the presetcorrection condition is satisfied. If the preset correction condition issatisfied, the processor 922 can perform image correction according tothe correction distortion coefficients.

In some embodiments, the image processor 922 is configured to capture animage of an object including straight line features through the cameralens 910, and adjust the size of the captured image to obtain the targetimage. In some embodiments, the image processor 922 is configured to, ifthe size of the captured image is smaller than a preset size threshold,magnify the captured image to a target size through interpolation, andif the size of the captured image is greater than the preset sizethreshold, scale down the captured image to the target size throughdown-sampling.

In some embodiments, the image processor 922 is configured to performedge detection on the target image to determine a border line in thetarget image, perform straight-line fitting on the determined borderline based on polynomial straight-line fitting to obtain a fittedstraight line, and calculate the first distortion metric value of theborder line corresponding to the fitted straight line and the correctiondistortion coefficients corresponding to the first distortion metricvalue.

In some embodiments, the image processor 922 determines a straight linesegment in the border line, calculates distance from correspondingpoints on the straight line segment to the fitted straight line andobtains the first distortion metric value according to the calculateddistances, and non-linearly optimizes the first distortion metric valueto obtain the correction distortion coefficients.

In some embodiments, the image processor 922 is configured to correctthe border line in the target image according to the initial distortioncoefficients or the correction distortion coefficients to completecorrection of the target image.

In some embodiments, the image processor 922 is configured to performedge detection on the second corrected target image to determine aborder line in the second corrected target image, perform straight-linefitting on the determined border line based on polynomial straight-linefitting to obtain a fitted straight line, and calculate the seconddistortion metric value of the border line relative to the fittedstraight line.

In some embodiments, the image processor 922 is configured to removeoutliers and determine a straight line segment in the border line, andcalculate distances from corresponding points on the straight linesegment to the fitted straight line and obtain the second distortionmetric value according to the calculated distances.

In some embodiments, the image processor 922 is configured to calculatea relative variation amount between the first distortion metric valueand the second distortion metric value. If the relative variation amountcalculated is smaller than a preset variation threshold, the correctioncondition is satisfied. Otherwise, the correction condition is notsatisfied.

In some embodiments, the image processor 922 is further configured toacquire preset initial distortion coefficients, or detect camera modelinformation and search for distortion coefficients corresponding to thecamera model information to configure the located distortioncoefficients as the initial distortion coefficients.

According to the embodiment of the present disclosure, a distortioncoefficient of an image can be determined comprehensively based onstraight-line fitting and a distortion metric value. This optimizes adistortion coefficient calculation manner and can obtain more accuratedistortion coefficients automatically and intelligently. The embodimentalso does not require an additional calibration board, and has a lowcost and is easy for users to use.

In the several embodiments provided in the present disclosure, it shouldbe understood that the related devices and methods disclosed may beimplemented in another manner. For example, the device embodimentsdescribed above are merely illustrative. For example, division of themodule or unit is merely division of a logical function, and division inanother manner may exist in actual implementation. For example, aplurality of units or assemblies may be combined or integrated toanother system, or some features may be omitted or not performed. Inaddition, the mutual coupling or direct coupling or communicationconnections displayed or discussed may be implemented by using someinterfaces, and the indirect coupling or communication connectionsbetween the devices or units may be electrical, mechanical or in anotherform.

The units described as separate components may be or may not bephysically separate, and components displayed as units may be or may notbe physical units, may be located in one position, or may be distributedon a plurality of network units. Some or all of the units may beselected according to actual needs to achieve the objective of thesolution of the embodiment.

In addition, functional units in the embodiments of the presentdisclosure may be integrated into one processing unit, or each of theunits may exist alone physically, or two or more units may be integratedinto one unit. The aforementioned integrated unit may be implemented ina form of hardware, or may be implemented in a form of a softwarefunctional unit.

When the integrated unit is implemented in the form of a softwarefunctional unit and sold or used as an independent product, theintegrated unit may be stored in a computer-readable storage medium. Thecomputer software product is stored in a storage medium, and includesseveral instructions used for causing a computer processor to performall or a part of a method consistent with embodiments of the presentdisclosure, such as one of the exemplary methods described above. Theforegoing storage medium includes any medium that can store programcodes, such as a USB flash drive, a portable hard disk, a Read-OnlyMemory (ROM), a Random Access Memory (RAM), a magnetic disk, or anoptical disc.

In some embodiments (for example, when only the distortion in onedimension is of concern, or when the distortion in one dimension isnegligible such that only the distortion in the other dimension needs tobe considered), instead of calculating the two distortion coefficientsk₁ and k₂ as discussed above, only one distortion coefficient k₁ or k₂may need to be calculated. The methods and apparatuses for calculatingthe one distortion coefficient are similar to those described above forcalculating both distortion coefficients, and thus detailed descriptionthereof is omitted.

The above descriptions merely relate to embodiments of the presentdisclosure, but are not intended to limit the scope of the presentdisclosure. Any equivalent structure or equivalent process variationmade by using contents of the specification and the drawings of thepresent disclosure, or directly or indirectly applied to other relatedtechnical fields, should be likewise included in the scope of thepresent disclosure.

What is claimed is:
 1. An image processing method comprising: correctinga target image based on an initial distortion coefficient to obtain afirst corrected target image; performing straight-line fitting on afirst border line in the first corrected target image to calculate afirst distortion metric value and a correction distortion coefficient;correcting the target image based on the correction distortioncoefficient to obtain a second corrected target image; removing outlierpoints on a second border line in the second corrected target image;performing straight-line fitting on the second border line with theoutlier points removed to calculate a second distortion metric value;detecting whether a preset correction condition is satisfied based on atleast one of the first distortion metric value or the second distortionmetric value; and applying, if the preset correction condition issatisfied, the correction distortion coefficient to subsequent imagecorrection to obtain better corrected images.
 2. The method according toclaim 1, further comprising, before correcting the target image:capturing an image of an object including straight line features; andusing the captured image as the target image or adjusting a size of thecaptured image to obtain the target image.
 3. The method according toclaim 2, wherein capturing the image of the object including straightline features includes: capturing a plurality of images; and analyzingthe plurality of images to determine the image of the object includingstraight line features.
 4. The method according to claim 3, whereinanalyzing the plurality of images includes analyzing the plurality ofimages at a same time using a same processing method.
 5. The methodaccording to claim 2, wherein adjusting the size of the captured imageincludes: magnifying, if the size of the captured image is smaller thana preset size threshold, the captured image to a target size throughinterpolation; or scaling down, if the size of the captured image isgreater than the preset size threshold, the captured image to the targetsize through down-sampling.
 6. The method according to claim 1, whereinperforming straight-line fitting on the first border line to calculatethe first distortion metric value and the correction distortioncoefficient includes: performing edge detection on the first correctedtarget image to determine the first border line in the first correctedtarget image; performing straight-line fitting on the first border linebased on polynomial straight-line fitting to obtain a fitted straightline; and calculating the first distortion metric value of the firstborder line relative to the fitted straight line and the correctiondistortion coefficient corresponding to the first distortion metricvalue.
 7. The method according to claim 6, wherein calculating the firstdistortion metric value and the correction distortion coefficientincludes: determining a straight line segment in the first border line;calculating distances from corresponding points on the straight linesegment to the fitted straight line; obtaining the first distortionmetric value according to the distances; and performing non-linearoptimization on the first distortion metric value to obtain thecorrection distortion coefficient.
 8. The method according to claim 1,wherein: correcting the target image according to the initial distortioncoefficient includes correcting a target border line in the target imagebased on the initial distortion coefficient, and correcting the targetimage based on the correction distortion coefficient includes correctingthe target border line in the target image based on the correctiondistortion coefficient.
 9. The method according to claim 1, whereinperforming straight-line fitting on the second border line with theoutlier points removed to calculate the second distortion metric valueincludes: performing edge detection on the second corrected target imageto determine the second border line in the second corrected targetimage; performing straight-line fitting on the second border line basedon polynomial straight-line fitting to obtain a fitted straight line;and calculating the second distortion metric value of the second borderline relative to the fitted straight line.
 10. The method according toclaim 9, wherein calculating the second distortion metric valueincludes: determining a straight line segment in the second border linewith the outlier points removed; calculating distances fromcorresponding points on the straight line segment to the fitted straightline; and obtaining the second distortion metric value according to thedistances.
 11. The method according to claim 1, wherein detectingwhether the preset correction condition is satisfied includes:calculating a relative variation amount between the first distortionmetric value and the second distortion metric value; and determiningwhether the relative variation amount calculated is smaller than apreset variation threshold to determine whether the preset correctioncondition is satisfied.
 12. The method according to claim 1, whereindetecting whether the preset correction condition is satisfied includesdetermining whether the second distortion metric value is smaller than apreset metric threshold.
 13. The method according to claim 1, furthercomprising: configuring, if the preset correction condition is notsatisfied, the correction distortion coefficient as the initialdistortion coefficient.
 14. A camera comprising: a camera lens; and animage processor configured to: correct a target image based on aninitial distortion coefficient to obtain a first corrected target image;perform straight-line fitting on a first border line in the firstcorrected target image to calculate a first distortion metric value anda correction distortion coefficient; correct the target image based onthe correction distortion coefficient to obtain a second correctedtarget image; remove outlier points on a second border line in thesecond corrected target image; perform straight-line fitting on thesecond border line with the outlier points removed to calculate a seconddistortion metric value; detect whether a preset correction condition issatisfied based on at least one of the first distortion metric value orthe second distortion metric value; and apply, if the preset correctioncondition is satisfied, the correction distortion coefficient tosubsequent image correction to obtain better corrected images.
 15. Thecamera according to claim 14, wherein the image processor is furtherconfigured to: capture an image of an object including straight linefeatures through the camera lens; and determine the captured image asthe target image or adjust a size of the captured image to obtain thetarget image.
 16. The camera according to claim 15, wherein the imageprocessor is further configured to: capture a plurality of images; andanalyze the plurality of images to determine the image of the objectincluding straight line features.
 17. The camera according to claim 15,wherein the image processor is further configured to analyze theplurality of images at a same time using a same processing method. 18.The camera according to claim 15, wherein the image processor is furtherconfigured to: magnify, if the size of the captured image is smallerthan a preset size threshold, the captured image to a target sizethrough interpolation; or scale down, if the size of the captured imageis greater than the preset size threshold, the captured image to thetarget size through down-sampling.
 19. The camera according to claim 14,wherein the image processor is further configured to: perform edgedetection on the second corrected target image to determine the secondborder line in the second corrected target image; perform straight-linefitting on the second border line based on polynomial straight-linefitting to obtain a fitted straight line; and calculate the seconddistortion metric value of the second border line relative to the fittedstraight line.
 20. The camera according to claim 19, wherein the imageprocessor is further configured to: determine a straight line segment inthe second border line with the outlier points removed; calculatedistances from corresponding points on the straight line segment to thefitted straight line; and obtain the second distortion metric valueaccording to the distances.